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The multiple perspectives of participants in a 
program review process at three schools of education were determined; 
along with the effect of these perspectives on enactment of program 
review policies. Participatincr schools were Emporia State University, 
which has a large teacher education program, and thi University of 
Kansas and Kansas State University, both of which are comprehensive 
research institutions with highly visible and politically sensitive 
schools of education. The objective was to understand the evolution 
of the program review process as "t occurred over time. A case study 
approach involved document review over an 11-month period; 
nonpartici,>ant observation; and in^.erviews with 48 university staff, 
Kansas Board of Regents members, and legislative staff who had 
participated in program review. Problems included: people holding 
positions in the universities changed over the period of the 
education review; school of education faculty, staff, and 
administrators tended to have a negative view about the review 
process, while university administrators, board members, and 
legislative staff tended to have positive views; participants were 
anxious about the review and reluctant to share informi.tion about the 
review process, and the political nature of the program review. 
Implications for research on program review are briefly considered. 
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Annual Meeting— February 20-23, 1986— Gunter Hotel 

§^n Antonio, Texas 



Kr'vj^ns A Cisv Study 

Willi" Ihvvr lids hcvn iiiiu li interest ree(Mitly in the Ucaure ami eifrctivc- 
ness oi pro^^rdiu reviews (Mingle, 19^1, Bdldrnige, Kemerer, cuni Crtu-n, 1982; 
Melch.ori, 1982), liltle hds been done to expioie the problems associated wit[i 
researching llw program review process itself. Conducting research on a 
process wlule [)eirig implemented confionts tlie researcher with tiie dilficulty 
oi documenting how people are changing, how their ideas are being levised and 
enlarged, how they are responding to uncertainty and anxiety, and how policies 
are being "adapted" to suit different contexts. The difficulty not msi - 
mountable, but portraying 1 ogi cs- i n-use demands a different discipline Crom 
presenting retrospective reconstructed logicj 

The program review pre v^ess in this study focused on the Schools of 
Education at three of the seven institutions under the control of the Kansas 
Board of Regents. Beginning in 1982, the Board had mi iateti a cyclical 
program review process wfiereby (^ach program would be reviewed every five 
years. Du(> to the press of the national education reform movement, education 
programs were reviewed m 1983, one year earlier than originally scheduled. 
The case study was conducted at three institutions witli 3 1 gn i f 1 1 an 1 1 \ different 
S(^opes and nu ssi on s. 

Botli tlie University of Kansas an<i Kansas State University W(^r(^ chosen 
because Ihey are comp reh(Mis i ve researcli institutions with highly visible^ and 
politically sensitive schools ol education. Kansas State UiMversity has as 
its primary mission extension and service to the state wlule the University of 
Kaiisas was noted for its research and tlu^ five-year extended t(M(lier education 
program. Kmporia State University was chosen because- it had a iaige teacher 
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♦ Mhicalion pro^rdm dn<i lias gaineu r^M^ogmnon for slroiig programs ni ihc jrcas 
ot a})[)Iir(] r(^s*^ircli to eciucation and iii-sprvice pi-ograms for tlie f)at)lic school 
serloi . 

Tlie purpose of Ihc study was to (Jelermine the multiple perspectives of 
tfie participants in the program review process and how these perspectives 
affected the enactment of the program review policies. Duiing the course or 
the research, several problem areas emerged which are the subject of this 
study. 

Problem S tatement 

In order to conduct research on a dynamic and fluid process, the researcher 
selected certain parameters within which the study would be conducted. These 
p2rameters included dimensions of time (the length of the study), dimensions 
of characters (who would be included) and dimensions of scope (what would be 
investigated) which were then used in a ''f reeze-f rame" approach to investigat- 
ing the program review process. That is, the study was limited to consideration 
of changes in program review policy during the period of time under investigation. 
However, while the researcher was taking the snapshot, the participants were 
changing their poses and economic, political and personal interventions were 
occurring which caused the study to increase in complexity. As Ian Mitroff 
said, "Ideas are like policies; policies, in turn, are like instruments. They 
either facilitate or they hinder our studies" (1980, p. 51A). 

The difficulty in conducting research on such a vigorous and impelling 
process as program review centers on two areas--the participants view of 
reality and the nature of policy itself. Participants in the program review 
process present a variety of backgrounds and experiences such as faculty, 
administrators, legislators and community leaders. Kach individual experi- 
ences the process from their own pose and tins determines their "reality" or 
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})(M spec t iV(» . H(Hjiis(^ tlu' pruioss is st^pdrdtt^d pliys i 1 1 y <mi(1 Ihv pl.wtTs .ire 
c ofisLdiit 1 y chdiiginx, tlir p.i rt i ciparit s jcL l.irgely witliout the knowl^'dgr ot tlic 
-Htioiis ot other pd!" 1 1 ( j [).in t s in the prot^ess and must rely c^n <i Lugi c s- i n-iisr 
philosophA' rdthei- thdii a recoiis t rncLeii logic. Later, the individuals leroii- 
strucled logK may reflect knowledge of c i rc unis tant e s unknown at the time ol 
act 1 on . 

Confounding the probl'^m of mullipJe perspective of the participants in 
the program reviev proc^^ss is the changing natur of policy itself. Policy 
statements once uttered or written do not stay as neat little axioms on which 
all future actions are based. Rather, they are implemei'ed, experienced^ and 
interpreted by the various participants in the process and metamorphoses into 
entirely new and different policy enactments. Thus, the objective ot this 
study was to examine the multiple perspective of the program review process 
under three constructs--policy-in-interition, policy-m-implementation, and 
pol icy- in- experience . 

Tins raises another problem associated with policy studies--that is, 
.determining what exactly constitutes policy in a given situation. frequently, 
the word "policy" has been nsed to describe different r^ncepts as noted by 
Klein and Murphy in 197J. They suggested that policy is better defined as a 
continuum moving from the very generalized statements about the purpose of an 
organization to specific statements regarding rules in an o rgaii i /a 1 1 on . 

In 198A Guba t[ieori/,ed tliat the word "pol u y" has multiple definitions 
and the type of definition applied in a situation "determines the kinds ot 
policy questions that are aoked, the kinds of policy-relevant data that are 
collected, the source of data tliat are tapped, the methodology that is used, 
and finally the policy products that emerge" (p. 63). in examining the liter- 
ature, Guba found eight b«sic definitions ot policy from winch he derived 
three policy types on which this study is ba.^ed. 
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The first type ol policy G.b. (198A) l.hrled .r, ,.o] u^'-in-.nLont ion .nu\ 
IS distmguishd hy assertion oi nitents or go.ls, staadmg goals, guides to 
discretionary actions and strategies to ;olve or ameliorate prohlems. 
I'ol^fV-JJ'-implpjnctitation was characterized by behaviors or activities 
demonstrated in the process of implementing policy Such behaviors or 
activities were define-^ by Guha (1984) as sanctioned behaviors, norms or 
conducts, and the cumulative output of the policy-making system. The third 
type of policy referreo to by Cuba was classified as the pol^i cy-jn-experMMice 
and represented the constructions or perceptions of the individuals as they 
experienced the policy enactment. The three const, ucts were applied to the 
program review process to determine what happens when participants change 
their minds, when political and economic considerations are applied on top of 
policy enactments and when the participants do not share a common reality but 
rather have multiple and fluctuating perspectives. 
Resea rch M e thds 

Since the primary concern of the study was to understand the evolution of 
the program review process as it occurred over time, the use of survey research 
was inappropriate. Rather the methodology used to gather data and information 
regarding the research inquiry evolved naturally out of the conditions of the 
study (Cuba and Lincoln, 1981). The conditions of this study were such that 
the participants, processes, and instiuitions were all dynamic and fluid 
Therefore, it was necessary vithin this study to .lisrover whatever patterns o* 
perspectives individuals involved with the program review perceived about the 
process. The research questions were fo, mulcted in order to invest I'oate the 
complexities of the process within its natural context and to derive under- 
standings about the behaviors from the subjects' own frame of reference (Rr^-dan, 
1982). 
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An inductive strcitcgy was cmpioytMi toi tin;- stui1y» bt^Ki'inin^ vv 1 1 h sjiccilu 
observations rc-garding the process ot proKrain re^vicw .in(i !)Uii(linK toward 
>^ori^M-al patterns. Tins is generally rolerred to as devi>loj)in^ grounded theory 
(Glaser an(i Strau^o, 19l)7) and according, to hincol . anci Guha (1982) is especially 
well-suited for naturalistic studies. For the purpose o{ this study, the case 
study method was the approach recommended by Lincoln and Guba (198S) because it 
allowed a reconstruction ol tlie respondents' constructions, l)uilt on the readers* 
tacit knowledge, and helped demonstrate the interplay lietween inquirer and 
respondent . 

In order to increase the likelihood that the widest range and scope of 
data would be uncovered, purposive or theoretica l Sompl ing was used. 
In naturalistic investigations, which are tied so intimately to 
contextual factors, tne purpose of sampling will most often be to 
include as much information as possible, in aP of its various 
ramifications and constructions; hence maximum variation sampling (a 
form of purposive sampling) will usually b*^ the sampling mode of 
choice. The object of tlie game is not to locus on tiie similarities 
that can be developed into generalizations, but to detail the many 
specifics that give the context its unique flavor (Lincoln and Guba, 
1985, p 201). 

Therefore, the sampling was not intended to be representative but to provide 
as wide a range o*' viewpoints as possible. The three mrtitutions chosen to 
be sampled in the study were selected because they repr(>sented critical cases. 
According to Patton (1980), using the sampling strategy of critical cases 
allows "logical generalization and maximum application of information to other 
cases because if it's tru.^ of this one case, it's likely to be true ol all 
otiiei cases" (p. If 
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T\w resejrtluM- oniployeMl tUirt' t Ln i (jiics ui the i jho sai(iy me t liodo 1 ogy 
UK hiding non-part ' c ipant ol)stM'vat i on , intrivieus ami document revie\; m 
galfuM-mg the data The doenme^nts anvi records uere examined and collected 
over an eleven-'iionth period vvhile tlu> re>searclter was on-sil at ont^ of the 
localions. Con.enl analysis was employed as llie method for the analysis oi 
documents and records. "Content arialysis is a research teciuuque for the 
objective, systematic, and quantitative description of the manifesi content of 
communication'' (Guba and Lincoln, 1981, p. 240). 

Interviews were conducte*' with lorty-eight indiv:duals including faculty, 
staff and administrators at the three institutions, selected members and 
former members of the Kansas Board of Regents, Board of Regents staff, and 
legislators who had participated in the program review. Since the basis of 
the inquiry was to search for the multiple perceptiouo regarding the process 
of program review, non-standardized, open-ended interviews were used. 

The coding procedure for this study was initially determined by the 
research questions. These categories included types oi policy statement^:, 
politicial, social or economic issues, process statements, ceitrality, quality 
and societal demand issues, and characteristics of internal and external 
models of review. As the study progressed, new categories were added and 
additional sorts were made. At the conclusion of tlie study, sixteen categories 
were used for the analysis of the data. 

Data analysis was performed simultaneously witli data collection by adapting 
a confimoiily used computer program Nultiplan to the task of sorting and retrieviiig 
the dat^. Tlie researcher found that by using tlie computer as a tool to managt>, 
retrieve and organize the data into a variety of patterns, the research was 
greatly facilitated. Conrad uud ReiUi^irz (198A) recognized this advantage and 
maintained that "by taking over many of the meclianical aspects of qualitative^ 



ERIC 



9 



7 



ERIC 



jii.ilysis, t.h(> roinpntn illows thr rt>stMr( h(M- to (i('V()t(> mere ol his or her 
(Mi(M-)^y to tlir 1 rit^M-prt^t Jt 1 or .uulytir work, which is more signilicant and 
rewarding" ( ]^)84, p. 9 ) . 
The Problems 

During the coarse of tr»e study, the researcher found that the niulliple 
perspectives of the participants and the nature of policy were prooL-matic to 
four areaj of conducting research into the progiam review process. The first 
area concerned the multiple perspective of the individuals who participated in 
the process. Because the process evolved over time, the actors m the process 
changed. For example, from the time the program review process started at one 
university until the education review was completed approximately two years 
later, the institution's president and chief academic officer changed, a new 
dean of education was appointed, and a new chief academic officer for the 
Board of Regents assumed office. Along with these organizational changes, the 
process was also characterized by a fragmented approach. Of the three institu- 
tions under study, only one institution had a committee charged with the 
responsibility of responding to the review from start to ''inish. Since this 
was the institution which experienced a high turnover in its top administration, 
this committee provided the or ly continuity for the review. The other two 
institutions followed a decentralized approach whereby one organizational 
level responded to one phase of the review and passed their report to another 
organizational level for response. In these circumstances it was difiicult to 
ascertain who was actually resDonding to the review and how much authority and 
responsibility were being exercised by the various parties. In one case, two 
administrative assistants exercised great degrees of control over one School 
of Educatioi's response. The problem for the researcher became one of identi- 
fying and seeking out the individuals who had such responsibi 1 it 1 es so tbat 
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Ihcir perspectives we»-e tle.u-ly uiuiei t jod . Kretjueiit 1 y , it was t tie staff 
members ol Vdrious deans and eliK^i cUadtMuie rjficeis Lliat really influenced 
ilie nuMUhM- in wliicli tlie policy was impl (Mi)(Mited . K(wievsing the iiil(>riial dccu- 
menLs, espe(ially inleriial nuMiio randiini , wert^ an important source tor this 
1 n forni.it ion . 

A second problem associated witli researching such a process as program 
review js allowing for t lie differing perceptions of the individuals in the 
process. The study found that those participants in the review process who 
were laculty members, staff or administrators withm the schools or colleges 
of education had a negative viewpoint about the review process. They indicated 
that a? far as they were concerned, nothing good came from the program review. 
In contrast, those participants who were university administrators or staff or 
Board of Regents members or staff, legislators or governor's staff maintained 
a positive view of the review process. 

Since the review process was viewed so differently by the various partic- 
ipants. It IS important for the research to allow for these multiple construc- 
tions and to reflect then m the findings. To search for a single reality 
under these circumstances is to deny the validity of these individual percep- 
tions. The researcher found in this case study that the individual felt very 
strongly about their viewpoints and further they as sumed that everyone else 
participating m tie review process shared their same viewpoints. Th(^refore, 
faculty members irate over the outcomes of the review assumed that their 
president was also outraged. This was not the case, and this limited study 
would suggest that governing boards and presidents should not assume their 
faculty and deans will perceive the value of program review and welcome it. 

B<-cause the review process was on-going and current actions could effect 
future decisions, there was a high degree of anxiety about the r^^view procc .s 
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ilseU. Tins .uixuay Ucis I r.nis 1 j tod into .u) mkIi vidual .Hid i nsl 1 1 »it i oria 1 
reluct.Hur to sh.u'e ujtorm.UKHi .ihout Ihv re»vu>w process and to bo candid ui 
individual romaiks. The resoarclur w.is of fee live in gathering Ino data only 
through the use ol key contacts who were able to acquire access lo documents 
and provide assurances to resp^.ndents . Witliout such cooperation and assistance, 
It is doubtful that the research could have been conipleteJ. Individuals were 
initially contacted through letters with follow-up telephone calls to secure 
intervjews. All individuals contacted, with Lhe exception of one, were willing 
to be interviewed. It was after the initial mtervi'-w, however, that individ- 
uals begun Ic. express concern about the inquiry, and follow-up interviews were 
often different in tone than initial interviews. Also during the initial 
interviews, the researcher a^ked for access to relevant documents and records. 
In all cases permission was granted for access to the documents and records 
but when the researcher would make follow-up visits to examine the documents, 
some documents ha^ disappeared and could not be located. The researcher found 
it best to gather documents the moment they were made available and to make 
copies * them for personal safe-keeping. Individuals felt so threatened that 
m several cases documents were either placed on the researcher's desk when no 
one was present or mailed to their office without identification as to the 
source of the documents. While this apprehension can certainly be appreciated 
given the political nature of the review process, it does complicate the 
research process and should be considered as a cha. acter i st i c of the process 
itself. 

The political nature of the program review is in itself a prob em to the 
researcher. The process of program review is a political process in the sense 
that it represents negotiations, compromises, adaptations and people making 
judgements and decisions about other people. It was jncumbant on the researcher 
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to know lu I on'y ds much jbout iiKiiVKiudi pt^rsptM 1 1 as possible hut dhoul 
the political aspects of the review, including the legislature and coordinating/ 
governing body attitudes, m order to effectively 'inderstand the process as it 
developed in the tield. 

While the process oi program review was v ^ed by board members, staff 
and administrators as c means of limiting the political effects of decision- 
making, the overall effect was to heighten the attention on decisions and, in 
part, acted to increase th-^ liticization of the process. Turf-baltles 
erupted between institutions which further increased the politics of the 
process. The program review process d:d not remove the politico from the 
decision-making arena. In fact, ,he .^^litical environment is not only 
inescapable, it is so pervasive one oh;>pl to expect its influence throuf> out 
the process. 

This IS why. in this instance, and in many other instances, policy-in- 
intent becomes a new crea ture--poli cy-in-implementat ion which ti*en changes 
into an entirely new creature--policy-in-cxperience . One cannot escape the 
infl'ience of policy because policy is the expression of the one or several 
value systenr.^ which operate in any given environment. 
Results and discussion 

Given the problems noted in the previous section, how then can research 
}nto such dynanjic processes s'^ch as program review be conducted? Traditional 
types of research which search for cause and etfect relationships and assume a 
single reality miss the fact that the actors in the processes do not share a 
common reality. In order to conduct effective inquiry i .cO dynamic processes, 
one must account for th?se multiple realities. Mitroff expressed this need 
when he said, "The realization has grown that observations are neither theory 
nor value free. One's values and one's theories influence what one sees and 
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how OIK* nilerprcts it. . . Ifitieevl, the prohlein is to show how different 
values and theoretical structures affect the collection and interpretation of 
data'^ (1980, p. 514), 

Based on the limited experience of one case study, one solutio^i to the 
problem is to examine the process from a wholistic sense looking for the 
larger context of the problem with all of its complexities. The '"luba (198A) 
theory of policy constructs provided ?n effective framework m which to 
understand the dynamics of the program review process. 

In the beginning the pol icy-m-intent statements, those statements repre- 
sented by Lhe official policy statements regarding program review and approved 
by the Board of Regents in December, 1982, provided the basis on which the 
process was structured. In general, all participants accepted those goals and 
objectives, and data were collected to support such goals. For example, in 
the early stages of the review process, consideration vas given to having all 
the institutions use the ETS surveys employed by the University of Kansas 
gr? late school. 

However, during the implementation stage of the process, various partici- 
pants began to make changes in the process, usually to achieve some other 
purpose than the ones stated in the original policy stateinents. Evidence of 
this was seen when the education reviews were moved up one year to gain added 
a tlenlior 

Nevertheless, it was only when the researcher interviewed the various 
participants in the process that the differences in pol i ry- in-exper lence were 
noted. It was found that each individual, based on his or her cwn experience, 
background and organizational perspcc* c, had assi;;*^.ed different values and 
meanings to the various activities in the process. Therefore, what the Board 
had intended as the goals of the pr >^ess were altered by both the intervening 
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policies ot the impl omenlal j mi sUjgo and llie individudl alliliidos ol the 
parlicipanls as they experieiKcd the policy actions. The process was perceived 
dilferently i iisl i tut i ona 1 1 y and individually. Th(* recognition and awareness 
that policy does change' over time is an important factor to be considered by 
both governing boards and administrations. The Cuba description of policy 
types <'ould suggest that policy will change from intent to implementat x'>n to 
experience and that such changes snould be expected by those involved in the 
program review process. 
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